Forward content_settings on write (open, pipe_file, put_file) by shcheklein · Pull Request #554 · fsspec/adlfs

shcheklein · 2026-07-06T02:51:04Z

Note: AI was heavily used to generate this. I reviewed the design principle and discussions on this topic across fsspec implementations.

Problem

AzureBlobFile only forwards metadata when committing a write — it drops content settings entirely. So there is no way to set content_type / content_disposition / cache_control / content_encoding when writing a blob through fs.open(..., "wb"), pipe_file, or put_file, even though the Azure SDK's commit_block_list / upload_blob / create_append_blob accept content_settings. This makes adlfs inconsistent with s3fs (extra kwargs → s3_additional_kwargs) and gcsfs (content_type / fixed_key_metadata), which set these inline in the atomic write.

Change

Add a content_settings parameter to open() (write modes), pipe_file(), and put_file(), applied atomically as part of the write commit (block, empty, and append blobs) — not a separate follow-up request.

To keep the Azure SDK out of the caller's imports, the documented/typed form is a plain dict; an azure.storage.blob.ContentSettings instance is also accepted for convenience.

import fsspec

with fsspec.open(
    "abfs://cont/report.pdf",
    mode="wb",
    content_settings={
        "content_type": "application/pdf",
        "content_disposition": 'attachment; filename="report.pdf"',
        "cache_control": "max-age=3600",
    },
    account_name="myaccount",
    connection_string=CONN_STR,
) as f:
    f.write(data)

# also works with pipe_file / put_file
fs.pipe_file("cont/x.pdf", data, content_settings={"content_type": "application/pdf"})

Notes

Scoped to content settings only. An earlier revision also let pipe_file/put_file override the hard-coded metadata={"is_directory": "false"}; that's unrelated to content settings (and metadata is already settable via fs.open(metadata=...)), so it was reverted.
Deferred, per discussion: no top-level content_type= kwarg for now — better introduced once the fsspec implementations converge on a shared interface (see filesystem_spec#916); a top-level field can be added alongside content_settings later.

Tests

Per-method tests (verified against azurite):

test_content_settings_open — streaming open("wb") + empty write
test_content_settings_append — open("ab") (append blob)
test_content_settings_pipe_file — pipe_file
test_content_settings_put_file — put_file
test_content_settings_accepts_object — a ContentSettings instance is accepted

Each verifies the settings round-trip via fs.info(...)["content_settings"]. Existing test_pipe_file_timeout / test_put_file_timeout updated for the new content_settings kwarg.

AzureBlobFile only sent metadata on commit, so content settings (content type/disposition, cache control, ...) could not be set when writing — unlike s3fs/gcsfs, which set them inline. Accept a content_settings argument on _open / AzureBlobFile and pass it to commit_block_list / upload_blob / create_append_blob so it is applied atomically as part of the write. Also let pipe_file / _put_file accept a metadata override instead of hardcoding {"is_directory": "false"}, matching the streaming write path and allowing metadata + content_settings together.

Replace the post-write set_http_headers step with inline content settings in the streaming commit, matching S3/GCS. This makes Azure writes atomic (no separate metadata request that a concurrent overwrite could misattribute) and removes the _finalize_write hook, its base definition, and the File.open finalize call. adlfs doesn't forward content_settings on write yet, so client/_adlfs_patch.py bridges it at runtime (mirrors fsspec/adlfs#554, no-op once released). AzureClient now maps WriteConfig to {content_settings, metadata} inline like the other backends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kyleknap · 2026-07-06T23:13:27Z

@shcheklein Thanks for the pull request! I think both exposing the content setting related values and allowing overriding of metadata across all write methods make sense functionally.

It's unfortunate that s3fs and gcsfs already diverge in being able to set content type, encoding, etc. I think if they were consistent, it would probably make sense to match their interfaces. So, matching SDK interfaces seems reasonable at this point.

We'll take a closer review at the PR later this week so we can make progress toward getting this merged in.

shcheklein · 2026-07-07T02:15:27Z

@kyleknap yes, agreed, I even found a few discussions:

fsspec/filesystem_spec#916

I was looking also into making this in a way that others can pick it up.

One practical thing we can do is to add at least content_type= (similar to what GCS has now). And then when all of the specs decide on the generic "additional_params" we can convert content_settings into that one and drop eventually. For now S3 and GCS are way too specific also, so hard to unify, and I'm not sure it makes sense making this decision here.

_adlfs_patch.py vendors adlfs's write logic (mirrors fsspec/adlfs#554); its copied empty-blob/append branches can't be exercised via azurite (adlfs's own source notes this), so omit it from coverage. Removed with the module once adlfs ships the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the vendored _adlfs_patch bridge (~120 lines mirroring adlfs internals) with Azure writes through azure-storage-blob's public upload_blob API, which sets content settings + metadata inline in one atomic upload. datachain no longer depends on fsspec/adlfs#554 or version-gating. Client.upload/File.open now delegate the actual write to overridable hooks (_write_object / open_for_write); the base keeps the pipe/stream logic for S3/GCS/local, and AzureClient overrides them to use upload_blob (buffering streams via a temp spool so the event loop never reads a loop-backed stream). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kyleknap

Thanks @shcheklein for the additional details! Yeah I agree that we should hold off adding content_type till we know we are likely to unify these parameters. Furthermore, it should not be an issue in supporting both a top-level content_type and one nested in content_settings at some point in the future.

I just left some suggestions on the PR. Overall, it looked good. Probably the biggest suggestion was that we tweak the content_settings parameter to also support providing these values as a dictionary to avoid forcing people from needing to import the SDK. We'd also document/recommend to users to use this dictionary format but we'd also accept (but not necessarily advertise) the ContentSettings object for compatibility use cases.

- Rename WriteConfig.extra → write_options to match the public kwarg and clarify it vs metadata (raw backend-native passthrough vs custom object metadata). - Remove the _build_write_config helper; construct WriteConfig directly (the helper was a pure passthrough after the rename). - Add a TODO linking fsspec/adlfs#554 in the Azure SDK write path. - Clarify open_for_write docstring (consumed by File.open). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Accept `content_settings` as a plain dict (documented form) in open/pipe_file/ put_file, coerced to ContentSettings internally; a ContentSettings instance is still accepted. Keeps the Azure SDK out of the caller's imports. - Keep the stored attribute private (`_content_settings`). - Split the content-settings test into per-method tests and add coverage for append (ab) mode, put_file, and passing a ContentSettings object. - Add CHANGELOG entries for the content_settings parameter and the metadata override in pipe_file/put_file.

_pipe_file/_put_file now always pass content_settings (None when unset) to upload_blob, mirroring metadata; update the exact-call assertions in test_pipe_file_timeout / test_put_file_timeout.

The metadata override in pipe_file/put_file is unrelated to forwarding content settings, so revert it (keep the hardcoded is_directory default) and drop its tests/changelog entry. content_settings is unchanged on open/pipe_file/put_file.

shcheklein · 2026-07-08T03:21:45Z

@kyleknap thank for the quick review! I've done another pass, cleaned things a bit and added more tests

* feat: object write metadata for File save/upload/open/export Add normalized, backend-agnostic object metadata to every File write path: content_type, content_disposition, cache_control, content_encoding, a custom metadata dict, and a raw write_options escape hatch. Threaded through File.upload, File.save, File.open (write mode), File.export, all subclass saves (Text/Image/Video/Audio/AudioFragment/VideoFrame/VideoFragment) and the save_audio/save_video_fragment helpers. A new WriteConfig carries the fields; each Client maps them to its backend via _write_kwargs/_finalize_write hooks: S3 -> s3_additional_kwargs; GCS -> content_type/metadata/fixed_key_metadata; Azure -> metadata inline plus a post-write set_http_headers for content settings (adlfs cannot set them inline); local ignores them. write_options is forwarded on S3 and rejected on GCS/Azure (their fsspec backends have no raw passthrough). Verified read-back across S3/GCS/Azure (moto/fake-gcs/azurite) x version_aware with unit mapping tests and cloud functional tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: trim write-metadata comments and test docstrings Drop comments restating the code, obvious hook docstrings, and a test helper docstring that repeated its return shape; keep only the non-obvious backend rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: reject write_options on backends that don't support it The base Client._write_kwargs forwarded cfg.extra, leaking the write_options escape hatch onto backends without an override (e.g. HfClient) despite the S3-only contract. Base now maps nothing and raises on write_options; only S3 forwards it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: set Azure content settings inline (atomic), via adlfs bridge Replace the post-write set_http_headers step with inline content settings in the streaming commit, matching S3/GCS. This makes Azure writes atomic (no separate metadata request that a concurrent overwrite could misattribute) and removes the _finalize_write hook, its base definition, and the File.open finalize call. adlfs doesn't forward content_settings on write yet, so client/_adlfs_patch.py bridges it at runtime (mirrors fsspec/adlfs#554, no-op once released). AzureClient now maps WriteConfig to {content_settings, metadata} inline like the other backends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: exclude temporary adlfs bridge from coverage _adlfs_patch.py vendors adlfs's write logic (mirrors fsspec/adlfs#554); its copied empty-blob/append branches can't be exercised via azurite (adlfs's own source notes this), so omit it from coverage. Removed with the module once adlfs ships the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document write_config / streaming-fallback in write docstrings Address PR review: note the new write_config param in save_audio / save_video_fragment Args, and that Client.upload streams bytes when the backend opts out of pipe_file (Azure). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: write Azure objects via azure SDK, drop vendored adlfs bridge Replace the vendored _adlfs_patch bridge (~120 lines mirroring adlfs internals) with Azure writes through azure-storage-blob's public upload_blob API, which sets content settings + metadata inline in one atomic upload. datachain no longer depends on fsspec/adlfs#554 or version-gating. Client.upload/File.open now delegate the actual write to overridable hooks (_write_object / open_for_write); the base keeps the pipe/stream logic for S3/GCS/local, and AzureClient overrides them to use upload_blob (buffering streams via a temp spool so the event loop never reads a loop-backed stream). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: address review — rename extra→write_options, drop helper - Rename WriteConfig.extra → write_options to match the public kwarg and clarify it vs metadata (raw backend-native passthrough vs custom object metadata). - Remove the _build_write_config helper; construct WriteConfig directly (the helper was a pure passthrough after the rename). - Add a TODO linking fsspec/adlfs#554 in the Azure SDK write path. - Clarify open_for_write docstring (consumed by File.open). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: re-trigger studio (run-level cancellation, unrelated to changes) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: Azure File.open append mode (don't overwrite) open_for_write buffered every write and wrote it via upload_blob(overwrite=True), which silently overwrote instead of appending for "a"/"ab" mode. Delegate append to adlfs's fs.open (append-blob semantics); keep the upload_blob buffer for overwrite/exclusive. Write metadata isn't supported on append (adlfs can't forward it there) and now raises instead of being dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: Azure exclusive ('x') write raises FileExistsError on conflict upload_blob(overwrite=False) raises azure ResourceExistsError; translate it to FileExistsError to match the Pythonic "x" mode convention (and adlfs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: avoid buffering whole File.open writes in memory on Azure open_for_write buffered every write in an in-memory BytesIO, risking OOM on large streamed writes. Now: when no write metadata is requested, use adlfs's native streaming (bounded, correct append/exclusive/update semantics); only when content settings/metadata are requested (which adlfs can't set on the streaming handle) buffer to a disk-spilling SpooledTemporaryFile and reject append/update modes. Also drop a non-idiomatic section-divider comment in the tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: larger spool copy buffer, fix upload docstring Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Simplify: centralize write_options reject, make _write_kwargs static _write_kwargs never used instance state and the write_options rejection was duplicated across base/GCS/Azure. Move the guard to WriteConfig.reject_write_options and make _write_kwargs a staticmethod, removing the object.__new__ test hacks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix misleading test name/comment for base-default write kwargs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Move client _write_kwargs tests into per-client test files test_write_config.py mixed WriteConfig dataclass tests with per-backend client _write_kwargs mapping tests. Move each backend's mapping/reject test to its own home (test_client_s3/gcs/local.py, test_client.py for the base default); keep only the WriteConfig dataclass tests in test_write_config.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kyleknap

This looks good to me. 🚢 Thanks @shcheklein!

shcheklein mentioned this pull request Jul 6, 2026

feat: object write metadata for File save/upload/open/export datachain-ai/datachain#1846

Merged

shcheklein marked this pull request as draft July 6, 2026 05:58

shcheklein marked this pull request as ready for review July 6, 2026 06:05

kyleknap added the enhancement label Jul 6, 2026

kyleknap self-assigned this Jul 6, 2026

kyleknap reviewed Jul 7, 2026

View reviewed changes

Comment thread adlfs/spec.py Outdated

Comment thread adlfs/tests/test_spec.py Outdated

Comment thread adlfs/tests/test_spec.py Outdated

Comment thread adlfs/tests/test_spec.py

Comment thread adlfs/tests/test_spec.py Outdated

Comment thread adlfs/spec.py Outdated

shcheklein added 3 commits July 7, 2026 16:00

Update timeout tests for content_settings kwarg on upload_blob

2626da3

_pipe_file/_put_file now always pass content_settings (None when unset) to upload_blob, mirroring metadata; update the exact-call assertions in test_pipe_file_timeout / test_put_file_timeout.

kyleknap approved these changes Jul 8, 2026

View reviewed changes

kyleknap merged commit e6c5cb4 into fsspec:main Jul 8, 2026
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Forward content_settings on write (open, pipe_file, put_file)#554

Forward content_settings on write (open, pipe_file, put_file)#554
kyleknap merged 4 commits into
fsspec:mainfrom
shcheklein:content-settings-on-write

shcheklein commented Jul 6, 2026 •

edited

Loading

Uh oh!

kyleknap commented Jul 6, 2026

Uh oh!

shcheklein commented Jul 7, 2026

Uh oh!

kyleknap left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shcheklein commented Jul 8, 2026

Uh oh!

kyleknap left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shcheklein commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Notes

Tests

Uh oh!

kyleknap commented Jul 6, 2026

Uh oh!

shcheklein commented Jul 7, 2026

Uh oh!

kyleknap left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shcheklein commented Jul 8, 2026

Uh oh!

kyleknap left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shcheklein commented Jul 6, 2026 •

edited

Loading

kyleknap left a comment •

edited

Loading